Log Retrieval Optimization

This EIE provides multiple methods to retrieve historical data, at least one of which is optimized.

The non-optimized way to retrieve historical data uses static parameters like start date, end date, and index to define data retrieval. The number of messages retrieved by the non-optimized method might be greater than necessary and might include records you've already retrieved in a prior poll. This method for retrieving historical data ignores available cache information because the information is irrelevant to the non-optimized request. If a cache preexisted the non-optimized retrieval, the cache remains unaffected. If a cache did not preexist the non-optimized retrieval, the latest record retrieved is used to populate the cache.

The optimized ways to retrieve historical data take advantage of a host-side transaction cache that keeps track of one or two items, depending on EIE:

• The most recently retrieved record ID
• The most recently retrieved timestamp

The record ID and/or timestamp values are cached for each successful and partially successful retrieval. The cached information provides a starting point for subsequent optimized retrievals. Optimization is achieved by using this cached information to more quickly locate the new record(s) to poll and to reduce the number of messages by retrieving only the most recent historical data. A transaction cache is maintained for each historical data group ordinal. Cached transaction information is stored in the Device Definition Service (DDS) transaction as data group elements.

See the following subsections below for more information:

• Optimized Retrieval Method(s)
• To Initiate a Get Latest Request Using a Data Group
• To Initiate a Get Latest Request Using a UIS Command
• To Clear the Cache for a Single Data Group/Ordinal
• To Clear the Cache for All Data Groups

For information about the data groups affected by historical data optimization, see Data Groups.

Optimized Retrieval Method 1: Start-Date Margin Retrieval

The first optimized way to retrieve log data is called a start-date margin request. This method works in conjunction with Start Date and End Date parameters and is automatically activated any time log data is requested one second after the most recent timestamp cached. This option does not require Get Latest activation.

For example, if the cached timestamp had a value of 03:00:00, a start date of 03:00:01 would trigger a start-date margin optimized retrieval. The last record successfully retrieved in the request is used to populate the cache. This ensures that a continuous, current, and relevant stream of log data is supplied with each new request.

Optimized Retrieval Method 2: Get Latest Retrieval

The second optimized way to retrieve log data is called a Get Latest request. To "get latest" means to get log data that has been saved by the field device after the most recent timestamp cached by the host; it is the data you do not already have. Get Latest uses the latest successfully retrieved record ID and timestamp combination as the point from which to collect log data in the next Get Latest request. This process ensures that, if properly used, you only get log data that has not already been retrieved in a prior poll. This method works in conjunction with Start Date and End Date parameters.

In the case of an initial poll (either the first poll of a device or after the cache is cleared), the latest successfully retrieved record ID and timestamp combination are stored in the cache so that Get Latest optimization is available for subsequent polls.

Three Get Latest scenarios follow; each assumes that you are using the Get Latest option.

Scenario 1

In this scenario, you request history data from a time window where the Start Date and End Date span the latest timestamp cached. The result is that only records that come after the latest timestamp and before or on the End Date (records 7 and 8) are retrieved and displayed. Records that come before the cache have already been retrieved. The cache is then updated to reflect the latest record retrieved (record 8).

Click the diagram below for more information:

Scenario 2

In this scenario, you request history data from a time window where both the Start Date and End Date are before the latest timestamp cached. The result is "No Data Available," which is returned because all available data has already been retrieved. The existing cache is left unchanged.

Click the diagram below for more information:

Scenario 3

In this scenario, you request history data from a time window where both the Start Date and End Date are after the latest timestamp cached. The result is that only history data for the window specified (records 8 - 12) is retrieved. The cached timestamp is set to the last record retrieved (record 12). Data between the cached timestamp and the Start Date is ignored (record 7). You might use this option because you know data immediately after a latest timestamp is unreliable for some reason, so you want to ignore it but still get the latest reliable data and bring the cache up to date.

Click the diagram below for more information:

To Initiate a Get Latest Request Using a Data Group

1. In the Device Definition Service (DDS), open the remote device you want to poll and click its Data Group page.
2. Open the relevant historical data group and click Get from RTU.
3. Select Get Latest.
4. Specify retrieval parameters, like Start Date, End Date, and Max Records to Read. Different combinations of these parameters produce significantly different results. Max Records to Read limits the number of records to be retrieved in one request.
5. Click OK.

To Initiate a Get Latest Request Using a UIS Command

1. In the Device Definition Service (DDS), open the remote device you want to poll and click its UIS Commands page.
2. Set up a UIS command to handle historical data retrievals.
   1. If available, activate the GetLatest command parameter (1=True, 0=False).
   2. If available, specify retrieval parameters, like Start Date, End Date, and Max Records to Read. Different combinations of these parameters produce significantly different results. Max Records to Read limits the number of records to be retrieved in one request.
   3. When you are finished defining the UIS command, click OK.
3. If polling recurs on a regular basis, schedule polls in the Master Scheduling Service (MSS).

To Clear the Cache for a Single Data Group/Ordinal

1. In the Device Definition Service (DDS), open the remote device you want to modify and click its Data Group tab.
2. Open the relevant historical data group and click Options.
3. Click Clear Tx Log Pointer Cache. If the Clear Tx Log Pointer Cache is not visible, no transactions exist.
4. If you are sure that you want to clear the cache for this data group, click Yes. The resulting Next Message ID is 0 and the Timestamp is 01 Jan 1970 00:00:00.000.

To Clear the Cache for All Data Groups

1. In the Device Definition Service (DDS), open the remote device you want to modify and click its UIS Commands tab.
2. Set up a UIS command to clear the cache.
3. At the Component Type drop-down menu, select Clear All Upload Log Tx Info [CLRLGTXA], then click OK twice.
4. On the UIS Commands page, highlight the newly added UIS command, then click Execute and Yes. Doing so clears the cache of every data group/ordinal combination for the selected remote device.